Introduction - Data Sources & File Formats
"Master Apache Spark and Big Data Engineering from first principles."
What You'll Master
Row-Oriented Formats
Reading CSV with header/inferSchema/delimiter options and single- or multi-line JSON with nested arrays and enforced schemas.
Columnar Storage
Parquet's columnar layout advantages, Snappy compression, predicate pushdown, and write operations.
JDBC & External Databases
Ingesting data from relational databases like PostgreSQL/MySQL via JDBC drivers, batch write controls, and custom query splits.
Real-World Case Study
A US Flight Delays case study covering Spark SQL temp views, high-performance distance queries, and custom Python UDF registrations.
Learning Path & Course Syllabus
Comprehensive guide on CSV reader options (header, inferSchema, custom delimiters) and multi-line parsing.
Ingesting data from relational databases (PostgreSQL/MySQL) via JDBC drivers, batch write controls, and custom query splits.
How to read single-line and multi-line JSON structures, handle nested arrays, and enforce schemas.
Columnar layout advantages, Snappy compression, pushdown features, and write operations.
An O'Reilly case study demonstrating Spark SQL temp views, high-performance distance queries, and custom Python UDF registrations.
A hands-on workbook applying CSV, JSON, Parquet, and JDBC ingestion patterns to real data-loading scenarios.
Scenario questions covering format tradeoffs, JDBC ingestion controls, and columnar storage internals.
What's Included in This Module
| Component | Coverage Details |
|---|---|
| Core Topics | Driver & Executor Architecture, Cluster Managers, Datasets |
| Practical Exercises | Interactive Hands-on Labs & Spark Tasks |
| Assessments | 1 Practical Assignment + 1 System Design Interview Quiz |